Multi - Armed Bandits in Discrete and Continuous Time

We analyze Gittins' Markovian model, as generalized by Varaiya, Wal-rand and Buyukkoc, in discrete and continuous time. The approach resembles Weber's modification of Whittle's, within the framework of both multi-parameter processes and excursion theory. It is shown that index-priority strategies are optimal, in concert with all the special cases that have been treated previously. 1. Introduction. A multi-armed bandit is a control model that supports dynamic allocation of scarce resources in the face of uncertainty [2, 3, 8, 15]. Each arm of the bandit represents an ongoing project and pulling arms corresponds to allocating resources among the projects. In a discrete-time model, arms are pulled one at a time and each pull results in a reward. This is in contrast to continuous time, where a more appropriate view is that of a resource (time, effort) which is to be allocated simultaneously among the arms while accruing rewards continuously. The goal is to identify optimal allocation strategies, and in this paper it is achieved for bandits with independent arms and random rewards, discounted over an infinite horizon. Specifically, we analyze Gittins's Markovian model [9, 8] in discrete and continuous time, as generalized by [18] and [13] in the spirit of [17]. The approach resembles Weber's [20] modification of [21] (see also [5–7]), within the framework of both multiparameter processes [12, 13] and excursion theory [11]. It differs from [6, 7], which take a martingale-based approach. The outcome is a rigorous proof that is shorter and, in our opinion, conceptually clearer than its predecessors, both in discrete time [18, 5, 3, 12] but especially in continuous time [6, 7, 13, 14]. Of interest also is the connection with general excursion theory [1]; see, for example, the index representation (33), which generalizes (4.3) in [11] from a Markovian setting. The continuous-time model is formulated and its solution presented in Section 2. One could view discrete-time bandits as a special case of continuous time, where rewards and information change only on a discrete set of predictable epochs. Nevertheless, Section 3 constitutes a self-contained treatment in discrete time: being short and accessible, it provides an introduction to the solution in continuous time, by highlighting main ideas that are not obscured by (unavoidable) technicalities. Properties of the index process are developed in Section 4 and used, in Section 5, to solve the multi-armed bandit problem.